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Thomas  Bellocci,  Parbati  Ray  and  Shimon  Y.  Nof 


ABSTRACT 


A  lab  experiment  has  been  performed  using  an  ERP  simulator  to  study  the  impact  of 
information  failure  on  the  results  of  a  company.  Two  scenarios  have  been  considered: 
correct  but  delayed  information,  and  wrong  information.  The  influence  of  the  length  of  delay, 
of  the  error  size,  and  of  the  dataset  concerned  by  the  failure  have  also  been  studied. 

It  follows  from  the  analysis  that: 

The  consequences  of  a  given  information  failure  depend  on  the  dataset  in  which  the 
failure  occurs. 

For  a  given  dataset,  information  failures  impact  depends  on  the  failure  type. 

The  influence  of  the  length  of  delay  depends  on  the  dataset. 

The  influence  of  the  error  size  depends  on  the  dataset. 

So  far  companies  employ  local,  speciahzed  solutions  that  are  too  restrictive,  or  not 
comprehensive.  The  experiments  presented  in  this  paper  justify  economically  the  use  of 
solutions  with  variable  assurance  in  ERP  systems.  They  also  provide  directions  for  the 
design  of  autonomous  agents  systems  to  handle  these  assurance  problems. 
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Appendix:  ANOVA  Results  and  Graphs. 


1.  Problem  Introduction: 


1.1.  Purpose  of  the  experiment: 

As  a  step  in  refining  the  assurance  requirements  survey  and  showing  the  variable  needs 
in  information  assurance,  experiments  have  been  conducted  with  an  ERP  software 
simulator-trainer  called  MICSS  (Management  Interactive  Case  Study  Simulator)  [16]. 
MICSS  was  developed  to  simulate  the  functioning  of  a  company  with  a  team-oriented 
view. 

In  our  previous  research  [I],  it  has  been  discovered  that  we  can  encounter  3  scenarios 
regarding  information  in  an  ERP  system.  A  data  item  can  indeed  be  correct,  correct  but 
delayed,  or  wrong. 

So,  we  have  decided  to  carry  out  a  set  of  experiments  to  study  the  potential 
consequences  of  information  failures  on  the  results  of  a  company. 


1.2.  Previous  research: 

A  class  experiment  involving  the  undergraduate  students  of  course  IE332  has 
been  done  earher.  This  has  provided  us  with  a  large  amount  of  data  that  has  been 
analyzed  [17].  The  measures  were  not  fully  reliable  to  carry  out  a  deep  statistical 
analysis.  Nevertheless,  it  has  shown  interesting  trends  that  have  encouraged  us  in 
organizing  our  own  experiment,  where  we  could  master  aU  the  parameters. 


1.3.  Hypotheses: 

The  hypothesis  of  our  team  experiment  was  that  the  performance  of  a  company  would 
significantly  different  in  the  case  of  delayed  and  wrong  information  than  in  the  case  of 
correct  information. 

Hq  =  The  performance  achieved  by  a  company  with  a  baseline  pohcy  is  similar 
to  the  performance  achieved  by  the  same  company  when  information  failures 
(delayed  or  wrong  information)  occur  in  the  baseline  pohcy. 

Hi  =  The  performance  achieved  by  a  company  with  a  basehne  policy  is 
significantly  different  to  the  performance  achieved  by  the  same  company  when 
information  failures  (delayed  or  wrong  information)  occur  in  the  basehne  pohcy. 
0.05  (a  95%  confidence  interval  to  prove  the  hypothesis.) 

if  p  val  <  0.05 ,we  can  conclude  with  95%  confidence  that  we  reject  the  null 
hypothesis  Hq 

To  verify  the  above  hypothesis,  the  data  was  analyzed  using  single  factor  ANOVA,  an 
analysis  tool  in  EXCEE. 

2.  Method: 


2.1.  Equipment: 

MICSS  (Management  Interactive  Case  Study  Simulator)  [16]  is  an  ERP  simulator  that 
has  been  developed  to  simulate  the  functioning  of  a  company  with  a  team-oriented  view. 

MICSS  has  four  views  of  a  company,  namely  Marketing,  Production,  Purchasing  and 
Finance.  Each  of  these  views  has  certain  pohcies,  which  combine  in  an  optimal  way  in 
order  to  be  profitable  for  the  company.  However  often  the  four  departments  of  the 
company  are  unable  to  communicate  properly  and  this  creates  discrepancies  in  the  pohcies 
developed  and  hence,  in  information  assurance. 

MICCS  enables  us  to  simulate  the  functioning  of  a  company  through  one  year. 


2.2.  Design  of  experiment: 

We  have  decided  to  study  4  factors  in  this  experiment. 

Factor  1 : 

Dataset;  with  4  levels:  Prices,  QET,  Batch  Size,  and  Order  Eevels. 

Factor  2: 

Failure  type;  with  2  levels:  “wrong  information”,  and  “delayed  informahon” 
Factor  3  (nested  in  “wrong  information”): 

Error  size;  with  2  levels  “value  doubled”,  and  “value  halved”. 

Factor  4  (nested  in  “delayed  information”): 

Length  of  delay;  with  2  levels  “4  months”,  and  “8  months”. 

So,  we  finally  had  17  scenarios  to  simulate: 

Fist  of  ah  the  scenarios: 


-Correct  information: 

(1)  Baseline  pohcy 

-Wrong  information: 

(2)  QFT  doubled 

(3)  Prices  doubled 

(4)  Batch  Size  doubled 

(5)  Order  Fevel  doubled 

(6)  QFT  divided  by  2 

(7)  Prices  divided  by  2 

(8)  Batch  Size  divided  by  2 

(9)  Order  Fevel  divided  by  2 


-  Delayed  information: 


(10)  QLT  delayed  4  months 

(11)  Prices  delayed  4  months 

(12)  Batch  Size  delayed  4  months 

(13)  Order  Level  delayed  4  months 

(14)  QLT  delayed  8  months 

(15)  Prices  delayed  8  months 

(16)  Batch  Size  delayed  8  months 

(17)  Order  Level  delayed  8  months 


2.3.  Metrics: 

To  assess  the  performance  of  the  company,  we  have  decided  to  record  the  Profit  and 
the  Due  Date  Performance  (DDP).  These  data  have  been  chosen  as  the  profit 
represents  how  the  whole  company  is  performing,  and  the  DDP  gives  an  idea  of  how 
well  the  company  is  organized. 


2.4.  Experimentation  Procedures: 

Wrong  information  scenarios 

A  dataset  of  the  basehne  policy  is  modified  (doubled  or  halved)  and  MICSS  is  mn  for  2 
months.  Then  the  data  is  corrected  and  MICSS  is  mn  by  periods  of  2  months  to  reach 
the  end  of  the  year. 

10  mns  of  one  year  are  performed  for  each  scenario. 

Delayed  information  scenarios 

A  dataset  of  the  basehne  pohcy  is  modified  (data-25%,  because  it  is  a  realistic  value 
that  can  be  encountered  in  the  functioning  of  the  company).  Then  MICSS  is  mn  for  4  or 
8  months,  by  periods  of  2  months,  depending  on  the  length  of  the  delay  we  are 
simulating.  Then  the  data  is  corrected  (i.e.  the  normal  value  of  the  dataset  in  the  baseline 
pohcy  is  released)  and  MICSS  is  mn  by  periods  of  2  months  to  reach  the  end  of  the 
year. 

10  mns  of  one  year  are  performed  for  each  scenario. 


3.  Results: 

The  observations  haven’t  been  analyzed  like  a  nested  design.  We  didn’t  need  ah  the 
information  given  by  a  nested  design  analysis.  For  simphcity  and  time  saving,  we  have  used 
single  ANOVAs  to  compare  each  time  two  different  scenarios. 

For  each  dataset,  the  following  comparisons  are  presented  in  Appendix: 

Dataset  delayed  4  months  /  Basehne  pohcy  (for  profit  and  DDP). 

Dataset  delayed  8  months  /  Basehne  pohcy  (for  profit  and  DDP). 


Dataset  wrong  half  /  Baseline  pohey  (for  profit  and  DDP). 

Dataset  wrong  double  /  Baseline  policy  (for  profit  and  DDP). 

The  datasets  are  presented  in  this  order:  Prices,  QLT,  Batch  Size,  Order  Level. 

Then,  the  influence  of  the  length  of  the  time  delay  and  of  the  difference  between  the  wrong 
and  correct  data  are  presented. 

Summary: 

Prices 

Fig.Al  -  Dataset  delayed  4  months  /  Baseline  policy  (for  profit  and  DDP). 

Fig.A2  -  Dataset  delayed  8  months  /  Baseline  policy  (for  profit  and  DDP). 

Fig. A3  -  Dataset  wrong  half  /  Baseline  policy  (for  profit  and  DDP). 

Fig.A4  -  Dataset  wrong  double  /  Baseline  policy  (for  profit  and  DDP). 

QLT 

Fig.AS  -  Dataset  delayed  4  months  /  Baseline  policy  (for  profit  and  DDP). 

Fig.Ab  -  Dataset  delayed  8  months  /  Baseline  policy  (for  profit  and  DDP). 

Fig.A7  -  Dataset  wrong  half  /  Baseline  policy  (for  profit  and  DDP). 

Fig.A8  -  Dataset  wrong  double  /  Baseline  policy  (for  profit  and  DDP). 

Batch  Size 

Fig.A9  -  Dataset  delayed  4  months  /  Baseline  policy  (for  profit  and  DDP). 

Fig.AlO  -  Dataset  delayed  8  months  /  Basehne  pohey  (for  profit  and  DDP). 

Fig.Al  1  -  Dataset  wrong  half  /  Basehne  pohey  (for  profit  and  DDP). 

Fig.A12  -  Dataset  wrong  double  /  Basehne  pohey  (for  profit  and  DDP). 

Order  Level 

Fig.AlS  -  Dataset  delayed  4  months  /  Basehne  pohey  (for  profit  and  DDP). 

Fig.A14  -  Dataset  delayed  8  months  /  Basehne  pohey  (for  profit  and  DDP). 

Fig.A15  -  Dataset  wrong  half  /  Basehne  pohey  (for  profit  and  DDP). 

Fig.Alb  -  Dataset  wrong  double  /  Basehne  pohey  (for  profit  and  DDP). 

Dataset  delayed  4  months  /Dataset  delayed  8  months 
Fig.AlV  -  Prices 
Fig.A18  -  QLT 
Fig.A19  -  Batch  Size 
Fig.A20  -  Order  Level 

Dataset  wrong  half  /  Dataset  wrong  double 
Fig.A21  -  Prices 
Fig.A22  -  QLT 
Fig.A23  -  Batch  Size 
Fig.A24  -  Order  Level 

Notations: 

“D”  means:  The  two  scenarios  give  significantly  different  results. 

“D  means  that  the  performance  with  information  failure,  for  profit 
or  DDP,  is  worse  than  with  the  basehne  policy. 


“D  +“  means  that  the  performance  with  information  failure,  for  profit 
or  DDP,  is  better  than  with  the  baseline  pohcy. 

“S”  means:  The  two  scenarios  give  significantly  similar  results. 


Table  1  summarizes  for  each  dataset: 

o  Which  information  failure  scenario  has  the  largest  impact  on  the  functioning  of 
the  company  (“1”  means  greatest  impact), 
o  Which  metrics  is  the  most  affected  by  a  failure  in  each  dataset, 
o  Whether  or  not  the  length  of  delay  has  an  influence  on  the  results, 
o  Whether  or  not  the  error  size  has  an  influence  on  the  results. 

A  complete  analysis  and  graphical  representation  of  these  results  can  be  found  in 
Appendix. 


Table  1  -  Summary  of  the  team  experiment  results. 


Dataset 

Prices 

QLT 

Batch  Size 

Order 

Level 

Impact  rank 

ing 

rong  half  3.  Delayed  8 

months  4. 

Delayed  4  months  1.  Wrong  double  2.  Wrong  half  3.  Delayed  8  months  4. 

Delayed  4  months  1 .  Wrong  half  Then  similar  for:  Wrong  double  Delayed  8 

months  Delayed  4  months  Similar  for  all  the  scenarios 

Metrics  sensitivity 

1.  Profit  2.  DDP 

Similar  for  profit  and 

:)DP  1.  DDP 

2.  Profit  Similar  for  profit  and  DDP 

Length  of  d 

elay 

Important 

Not  important 

Not 

important  Not  important 

Error  size 

Important 

Important 

Important 

Not 

important 


4.  Conclusions  and  Discussion: 


4.1.  Impact  graphs: 

Impact  graphs  summarize  the  impact  of  each  information  failure  type  by  dataset  (Fig. 
l.a  and  l.b).  The  relative  differences: 

a.  (Profit  with  information  failure  -  Profit  with  basehne  pohcy)  /  (Profit  with 
baseline  pohcy) 

b.  and:  (DDP  with  information  failure  -  DDP  with  basehne  pohcy)  /  (DDP  with 
basehne  pohcy) 

are  represented  respectively  in  Fig.  l.a  and  l.b. 

These  differences  are  shown  using  levels:  [>  70%;  35  to  70%;  5  to  35%;  +/-  5%;  -5  to  - 
35%;  -35  to  -70%;  <  -70%] 


The  fohowing  notations  are  used  in  Fig.  l.a  and  l.b: 


D4:  scenario  with  information  delayed  4  months 
D8:  scenario  with  information  delayed  8  months 
Wh:  scenario  with  information  wrong  half 
Wd:  scenario  with  information  wrong  double 


4.2.  Conclusions: 

1/  Some  datasets  are  more  sensitive  than  others.  For  example  the  consequences  of  a  problem 
concerning  Prices  is  much  more  serious  and  long  lasting  than  when  it  concerns  QLT.  We 
can  rank  the  datasets  that  have  been  tested  by  decreasing  sensitivity:  Prices,  QLT,  Batch 
Size,  Order  Level. 

2/  Datasets  have  different  characteristics  that  make  them  more  sensitive  to  a  specific  type  of 
information  failure.  For  example,  a  delay  of  8  months  has  a  large  impact  on  Profit  when  it 
concerns  Prices,  but  no  real  impact  when  it  concerns  QLT. 

3/  Profit  is  very  sensitive  to  information  failures.  DDP  react  more  slowly  and  need  long  lasting 
and  large  errors  to  be  modified. 


4/ 

5/ 

6/ 


The  importance  of  information  failure  has  been  proved. 

The  importance  of  the  length  of  delay,  and  of  the  error  size  has  been  proved. 

We  have  seen  that  different  scenarios  can  have  very  different  consequences.  A  targeted 
security  solution  can  then  be  designed  to  prevent  the  most  serious  cases  first. 
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APPENDIX 


Fig.Al  -  Prices;  Dataset  delayed  4  months  /  Baseline  policy  (for  profit 


and  DDP). 
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Observations: 

For  profit:  during  the  4  months  of  delay,  the  performance  is  worse.  Then  when  the 
information  is  corrected  (return  to  the  basehne  pohcy)  the  company  follows  the  same 
evolution  than  with  the  correct  scenario,  but  the  gap  due  to  the  delay  cannot  be  filled. 
For  DDP:  There  are  shght  consequences  that  can  be  easily  filled  when  the  information  is 
corrected. 


Fig.A2  -  Prices;  Dataset  delayed  8  months  /  Baseline  policy  (for  profit 
and  DDP). 
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Observations: 

For  profit:  during  the  8  months  of  delay,  the  performance  is  worse.  Then  when  the 
information  is  corrected  (return  to  the  basehne  pohcy)  the  company  follows  the  same 
evolution  than  with  the  correct  scenario,  but  the  gap  due  to  the  delay  cannot  be  filled. 


For  DDP:  There  are  slight  eonsequenees  that  can  be  easily  filled  when  the  information  is 
corrected. 

Fig.  A3  -  Prices;  Dataset  wrong  half/  Baseline  policy  (for  profit  and 
DDP). 
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Observations: 

For  profit:  The  information  is  wrong  only  during  the  first  2  months.  But  when  the 
information  is  corrected  (return  to  the  basehne  pohcy)  the  company  doesn’t  foUow  the 


same  evolution  than  with  the  correct  scenario.  In  this  case,  wrong  information  has  long- 
lasting  consequences. 

For  DDP:  There  are  not  major  consequences. 


Fig.A4  -  Prices;  Dataset  wrong  double  /Baseline  policy  (for  profit 
and  DDP). 
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Observations: 

For  profit:  The  information  is  wrong  only  during  the  first  2  months.  But  when  the 
information  is  corrected  (return  to  the  basehne  pohcy)  the  company  doesn’t  foUow  the 
same  evolution  than  with  the  correct  scenario.  In  this  case,  wrong  information  has  long- 
lasting  consequences. 


For  DDP:  There  are  not  major  consequences. 


Fig.A5  -  QLT;  Dataset  delayed  4  months  /  Baseline  policy  (for  profit 
and  DDP). 
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Observations: 


For  profit:  Consequences  last  2  months  after  the  return  to  the  basehne  pohcy.  Then  the 


company  follows  the  same  evolution  than  with  eorrect  information  without  any  gap. 
For  DDP:  There  are  major  eonsequenees  during  the  4  months  of  delayed  information. 
But  the  eompany  reeovers  as  soon  as  the  information  is  eorreeted. 


Fig.A6  -  QLT;  Dataset  delayed  8  months  /  Baseline  policy  (for  profit 
and  DDP). 
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Observations: 

For  profit:  The  performance  of  the  company  is  affected  during  the  8  months  of  delayed 
information.  Then,  after  returning  to  the  baseline  pohcy,  the  company  follows  the  same 
evolution  than  with  correct  information  without  any  gap. 

For  DDP:  There  are  major  consequences  during  the  first  4  months  of  delayed 


information.  Then  the  performance  is  not  significantly  different  from  the  basehne  scenario 
even  if  it  looks  slightly  worst. 


Fig.A7  -  QLT;  Dataset  wrong  half  /  Baseline  policy  (for  profit  and 
DDP). 
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Observations: 

For  profit:  After  returning  to  the  baseline  policy,  the  company  doesn’t  foUow  the  same 
evolution  than  with  correct  information.  The  slope  is  smaller.  There  are  long-lasting 
consequences. 

For  DDP:  There  are  major  consequences  that  are  long  lasting.  At  the  beginning  the 
company  receives  a  lot  of  orders  as  it  provides  products  at  the  same  price  but  with  a 
QLT  half  of  the  baseline  policy.  Therefore  the  DDP  decreases  as  the  company  cannot 


satisfy  all  the  orders  on  time.  So  the  amount  of  orders  decreases  and  the  DDP 
increases.  This  explains  why  the  profit  is  finally  smaller  in  period  6. 


Fig.AS  -  QLT;  Dataset  wrong  double  /  Baseline  policy  (for  profit  and 
DDP). 
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Observations: 

For  profit:  After  returning  to  the  baseline  policy,  the  company  doesn’t  foUow  the  same 
evolution  than  with  correct  information.  The  slope  is  smaller.  There  are  long-lasting 


consequences. 

For  DDP:  There  are  major  consequences  that  last  even  after  returning  to  the  basehne 
policy.  But  finally  the  gap  is  filled. 


Fig.A9  -  Batch  Size;  Dataset  delayed  4  months  /  Baseline  policy  (for 
profit  and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  delayed  information. 
For  DDP:  There  are  no  consequences  of  delayed  information. 


Fig.Al  0  -  Batch  Size;  Dataset  delayed  8  months  /  Baseline  policy  (for 
profit  and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  delayed  information. 
For  DDF:  There  are  no  consequences  of  delayed  information. 


Fig.Al  1  -  Batch  Size;  Dataset  wrong  half/  Baseline  policy  (for  profit 
and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  wrong  information. 
For  DDP:  There  are  no  consequences  of  wrong  information. 


Fig. All  -  Batch  Size;  Dataset  wrong  double  /Baseline  policy  (for 
profit  and  DPP). _ 
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Observations: 

For  profit:  There  are  no  consequences  of  the  wrong  information  that  occurred  during  the 
first  2  months,  but  they  appear  only  at  the  end  of  the  year  in  period  6.  This  is  certainly 
due  to  the  behavior  of  the  DDP. 


For  DDP:  There  are  major  consequences  of  wrong  information.  It  evolves  in  parallel 
with  the  baseline  scenario,  and  it  never  reaches  the  same  level  of  performance.  The 
consequence  of  this  low  DDP  is  a  smaller  profit  in  period  6. 


Fig.AlS  -  Order  Level;  Dataset  delayed  4  months  /  Baseline  policy  (for 
profit  and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  delayed  information. 
For  DDP:  There  are  no  consequences  of  delayed  information. 


Fig.AM  -  Order  Level;  Dataset  delayed  8  months  /  Baseline  policy  (for 
profit  and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  delayed  information. 
For  DDF:  There  are  no  consequences  of  delayed  information. 


Fig.AlS  -  Order  Level;  Dataset  wrong  half  /  Baseline  policy  (for  profit 
and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  wrong  information. 

For  DDF:  There  are  no  consequences  of  wrong  information.  The  strange  pattern  that 
can  be  seen  in  period  5  and  6  for  “wrong  half’  is  explained  by  some  outlaying  measures, 
which  modify  the  average  shown  in  the  graph  but  not  the  statistical  analysis. 


Fig.A16  -  Order  Level;  Dataset  wrong  double  /Baseline  policy  (for 
profit  and  DDP). 
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Observations: 

For  profit:  There  are  no  consequences  of  wrong  information. 
For  DDP:  There  are  no  consequences  of  wrong  information. 


Fig.Al?  -  Prices;  Dataset  delayed  4  months  / Dataset  delayed  8 
months 
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Observations: 

For  profit:  Length  of  delay  has  influence  on  the  performance. 
For  DDP:  Length  of  delay  has  no  influence  on  the  performance. 


Fig.Al  8  -  QLT;  Dataset  delayed  4  months  /  Dataset  delayed  8  months 
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Observations: 

For  profit:  Length  of  delay  has  no  influence  on  the  performance. 

For  DDF:  Length  of  delay  has  no  influence  on  the  performance. 

Fig.A19  -  Batch  Size;  Dataset  delayed  4  months  /Dataset  delayed  8 
months 
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For  profit:  Length  of  delay  has  no  influence  on  the  performance. 
For  DDF:  Length  of  delay  has  no  influence  on  the  performance. 


Fig.A20  -  Order  Level;  Dataset  delayed  4  months  /  Dataset  delayed  8 
months 
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Observations: 

For  profit:  Length  of  delay  has  no  influence  on  the  performance. 
For  DDP:  Length  of  delay  has  no  influence  on  the  performance. 


Fig.A21  -  Prices;  Dataset  wrong  half /Dataset  wrong  double 
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Observations: 

For  profit:  Error  size  has  a  great  infiuence  on  the  performance. 

For  DDP:  Error  size  has  infiuence  on  the  performance  in  the  short  term  only. 


Fig.A22  -  QLT;  Dataset  wrong  half/  Dataset  wrong  double 
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Observations: 


For  profit:  Error  size  has  a  great  infiuence  on  the  performance. 


For  DDP:  Error  size  has  a  great  influence  on  the  performance. 


Fig.A23  -  Batch  Size;  Dataset  wrong  half/  Dataset  wrong  double 
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Observations: 


For  profit:  Error  size  has  infiuence  on  the  performance  in  the  long  term. 
For  DDP:  Error  size  has  a  great  infiuence  on  the  performance. 


Fig.A24  -  Order  Level;  Dataset  wrong  half  /  Dataset  wrong  double 
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Observations: 

For  profit:  Error  size  has  no  infiuence  on  the  performance. 

For  DDP:  Error  size  has  no  infiuence  on  the  performance.  The  strange  pattern  in  period 
5  and  6  for  “Wrong  half’  is  explained  by  outlaying  measures,  which  modify  the  average 
shown  on  the  graph  but  not  the  statistical  analysis. 


